Human body 3D reconstruction in multiview soccer scenes by depth optimization

In soccer matches, 3D reconstruction is a main part of many applications including free-viewpoint broadcasting, match analysis, augmented reality, and refereeing examination. The main challenge of 3D reconstruction in soccer scenes is the human body reconstruction. Although 3D reconstruction methods have been improved to a high extent in controlled condition, still there are lots of uncovered issues in outdoor scene reconstructions with uncontrolled conditions. In this paper, a view-dependent depth optimization method is presented which addresses two of these issues in soccer scenes. These issues include inaccurate camera calibrations and limited number of input cameras. This method will optimize the proposed energy functions to find the optimal depth image related to each input camera. For computing the initial depth images, a visual hull computation technique is used. An energy function based on depth images is defined which includes four energy terms. Two new energy terms including silhouette term and pose term are introduced in this function. The energy function is optimized with the α-expansion algorithm to find the optimal depth image. This optimization is done for each camera, separately. Finally, the 3D model is reconstructed with the resulted depth images. Experimental results prove that compared to previous methods, better quality 3D models are achieved. Also, as an application, free-viewpoint videos are created. The experimental results show the efficiency of the proposed method.